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creation of nev 
report contains 

encountered first-han^ by evaluation *pra,Qtitioners in state 
departments of education CSEIs). Jts latent is' to make available 
practitioner statements so that further developmental irork may be 
arounded in first-hand accounts. Practitioners yere requested to 
Incrude, vh^re appropriate, material concernlngi methodological 
problems and Mthods ^requiring improvement: ^proposals for nev 
methods, and any problems involved in the development of ^uch: 
materials jand training needed to improve methods: personnel, 
management, planning, resource and interagency communication 
problems. The case descriptions are varied both iom style and the 
problem cases they divulg^^- Included ar^ such vide^ranging topics as: 
SEA and school ^district autonomy: obtaining and allocating funds for 
eValuatlon purposes: • the evaluation need-s of different audiences: , 
data collection probJ.ems and qua'lity control: the iiaximization of 
evaluation, utilization-r (iuthor/iEP) , * * 



» af a series* of reports conce^aed 
ilTOtion 



with the 

evai^tion methodologies for use in education, this 
a collection of thirteen^ brief statements of problems 



r 
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PREFACE 



The Res^arc^^ on Evaluation. P^rogram is a Northwest Regional - 
.Educational Laboratory project of research, aevelopment, testing, 
•and tfaining designed to create new evaluation methodologies for 
^^se m education. This document 'is one of a series of papers and 
reports produced by program staff, visiting scholars, adjunct 
holars, and project collaborator s--all members of a cooperative 
etwork of colleagues working on the developirfent of nev^ ' 
methodologies. - ♦ 

* > 
What kinds of problems do evaluators encounter -m the course of 
their work? This report contains a collection of thirteen brief 
statements (2-9 pages) of probikems encoun-tered by evaluation 
practitioners in sta^e-^epartments of edtjcat-ion. These problem 
case descriptions, prepared by practitioners tnemselves,. provide 
msignt into the difficulties of state level evaluation practi^. 

Nick L. Smith, Editor 
/ Paper and Report Series 
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, ' . PROBLEM CASE DESCRIPTIONS 

i Introduction ^ * % 

.What kinds bf problems do evaluator^ encounter in the course 
of their work? This report contains a collection of thirt^n , 
such problems as repo/ted by evaluation practitioners^ in state 
education agencies (SEAs) . These cases were assembled by Dr. 
Darrel N. Caulley of the Reseajc<:h on Evaluation Program so that 
prograun staff would have a better understanding ,of the '* 
difficulties encountered by st4te department evaluators. 
Assumptions ab9ut the nature of evaluation practice, its pr'oblems 
and constraints, underlie much of the wOrk of tb,e program. These 
collected practitioner staten^ents enable program' staff to test, 
-the validity of staff views of practice and to ground further 
work in first-hand accounts of evaluation practice, 

A number of evaluators irr state ^departments of education ^wer^ 
paid' a nominal- fee to prepare brifef (2-9 pages) statements of *a 
problem they^had enibuntered in their woric, 'They were aske;3 to 
descri<)e an actual problem in sufficient detail so that others 
could understand 'the natui;e of the problem, its context and . . 

* implications. , The writers were allowed to select any problem 
' they wished and to presentXit i^ the form they thought most 
useful. (Of course, names, dates, etc. were altered to insure 
anonymity.) To aid their efforts, writers were' sent samp^^e ' 

' problem case descriptions obtained from business education 
materials. The following li^t of possible problem topics was 

.^Xbo provided:./ 

i 



a. What mfethodological problems do SEAs currently have? 

b. What new methods are needed? 
Which methods need^ to be improved? 



c . 
d. 



What problems are there m the develoyment of new 
methods? 



e. What problems are there in. the implementation . of new 
* methods? ' f 

f. * What kinds of itiaterials and training systems are 

needed to help SEAs improve methods? 

4 

g. Wl^t serious personnel/staff >jig problems does your 
SH^A evaluation unit have? 

h. What serious management problems does your SEA 
evaluation unit have? 

1. What serious planning problems does your SEA 
/ evaluation' unit have? ' ^ . 

3. What important resout/:es problems does your sfeA 
evaluation jjnit have? / 



What important reporting problems does y^ur SEA 
evaluation unit have? . * / 

What important interagency cbnnnunicajl^ion problems 
does your SEA "Valuation unit have?/ 



The thirteen problem stat^ents whic)i follow vary 



considerably in topic, length, 'aijd 
the "following indivi'dd^ls who prov 
descriptions: . < 




Mary Anri Awad 
Bill Burson 
Alex Hazelton 
Jerry Hutchinson 
Thomas Kerins 
Ann Kraetzer 
George Malo 
Claudia M%rkel^;JK<ell/r 
Lyn Nachman / 
Michael Plog / 
Norman Stenzel / 
Donna Van Kirk/ 



t. We are gtateful to 
these problem case 



Ne^ Yo 




York Department of Education 
lifornia Department of Education 
^Alaska Departme'nt of Education 
Mississippi Department of Education 
Illinois Office of Education 
Colorado Depa^bment; of Education 
Tennessee Depar tmeHt^of Education^ 
New Jeru^ Departm^N^ of Education 
Minnes^a Department of Education 
Illinois Office of Education 
Illinois Office of Education % 
Washington Depattroent of Education 



Dr,. Caulley originally intended to supplement this collection 
of case descriptions wi^th, additional analysis and commentary. 
Due to a seriouq^ and lengthy illness, hdwever, he has been unable 
to do so. Hopefully, he will be able to retcTrn to this task in a 
fevT months. 'In the meantime, these problem^ statements have been 
gathered he|Cre^ for use by program staf.f. 

Nick L. Smit^h 



Problem Case Description No. 1 
Nonpublic Student Auxiliary Services Program 



J 



Background . * • • 

Why Mary can*t read and wby Johnny can*t add are serious 
concerns Facing educators at all levels — local, state and 
national. 

A state depart^nt of education in one of the^ greater ' 
metropolitan are^s of the nation is develcJping and implementing 
signifiq^nt remediation and compensatory programs to deal with 
the large number^ of its students .falling below the 
state-established minimum basi<? skills performance levels. By 
state mandate all of those^ stydents falling below the state 
standard must be served -b^ a remedial program which can be 
funded, in^ full or in part, by local, state and fedeirail funds or 
a combination of these. As can be e^tpected,' the ^tate^ department 
of education implemerits a very sizeable Title I prpgram which 
serves both public and qonpublic students. Approximately 77 - 
million dollars ar^ allocated to the state by the federal 
government every year for- the provision of Title I programs and 
projects. 

< • The State Legislature has also made a commitment to provide 
^remedial programs to the ^state's students through the allocation 
of about 68 million dollars ftr the State Compensatory' Education 
Program. The State Compensatory Education program is 
admini'stered by the State Department of Education and is aimed^'at 
the public sector. In 1977, the State Legislature alsp passed 
two laws with a funding package of about 9 million dollars which 
provided auxiliary services and general' services for nonpublic 
students. This -leg-islat ion was a significant step forward -in 
terms of the allocation of public monies to service the nonpublic 
sector. At the hear$^f the issue is the serration of church 
and state"; ^ ♦ f 



r 
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As can be seen from th^ previous discussion, ,this state's 
budgetary commitment to providing compensatory -services to its 
students is in the neighborhood of 160 million dollars. Of 
course, from a policy and decision making level, both state and 
national, the question' arises, "Is the program working?" ; th^tx is 
to say, "Are the children learning more as a result of the 
program?" The^e accountability and, evaluation questions, as well 
as others, gave rise'to the. development of the Title I Evaluation 
and Reporting System (TIERS) at the national level through 
pressure .from the U.S. Congress. 

The TIERS system attempts to formalize data collection and 
reporting across the states through the implementation o& thr^e 
outcome evaluation models 6r designs coupled with the gathering 
of other program and process data. ^ 

( 



( The three outcome evaluation models or designs are as follows 



Model A: the norm-referenced model. 
Model B: the control-group models 
Model C: the special regression mode,l. 

. All three models are designed^ to be used with any valid and 
reliable norm-referenced test or « iter lon-ref erenced test. 
Additionally, 4ach of the models requires both pre-testing and 
post-testing and imposes ^ some special conditions and restrictions 
on the testing itself. The three models each provide data on an 
observed post- treatment performance* measure and an estimate of 
what that performance would have been without the program (i.e., 
without the treatment) 

This state viewed the models proposed and then mandated in 
1979, for Title I^programs as a viable method to look at ^ 
evaluation for all of its basic skills preventive and remedial 
programs in the state regardless of funding source. Th^ state 
took a bold step and mandated the use' of the Title I Evaluation 
and Reporting System C^ERS) fo;: all of its compenfiratory programs 
operating in the state. The state^haS been successful in the 
implementation of TIElJfi for its Title I programs and for its 
State Compensatory ^Education in the public sector.- Plans ard 



currently underway to augment the basic-output design with other 
types of pcogram ana process data. 

Issue , . • 

At the co're^of this issue, however, is -the state's role in 
the J5lanain9v ^development , and implementation, of an evaluation 
system including TIERS for the compensatory educTation services 
provided under the Nonpublic Student Auxiliary Services Program 
(see Figure IJ . . ' 

The issue is a complex one, since it not only deals with (1) 
the issue of state control and governance in terms of the 
9eparation of church and state, but also with (2) philosophical 
concerns which suggest that'^if an agency accepts monies from the 
state that agency must be accountable to the state for those 
monies apd musX- be governed by the implementation rulea for the 
use of those monies. Added to the governance issue is (3) the 
'issue of program planning and evaluation. At the state' level, 
one needs to know if the program is working, and indeed, if 
program services are reaching the intended audiences* At 
present, the State Department of Education is at a loss ^o be 
able to say anything much beyond 'the total ^utlay of monies by 
category by school district. 'State monitoring of the program is 
virtually non-existent. Again, by default .<or at leat post-hoc) , 
p;:ogram Valuation plajuning may give su'bstSince and sh^pe to the 
overall planning, implementation, and delivery of the particular 
program inviuestion. 





FKiURE ^ 



MAJOR STATE BAS4C SKILLS' HRtVEN 
AND REM-DiA|^ PROGRAMS AflO FUNQI 



Eviilujitlon Probifim 
A/ej 



$77 MilHon 
(federil fun^t) 



State ^Com^nsA to ry Edufation^'' 
« Progrlcn 
.i^S Million 



Nonput>l)c^tude^^^tv^}<iliary 
Services Program 
$9Millicfn 
(stats funds) 



Targetad to Ihd puljlic and noiif^bliL ^ecior 
Stattt has direct govttrnance.'pldru^ingc and 
ftionitoring responsibilities lor program and b^d9tit 



Targeted to the public Stitor 

State has direct governance, planning and rnonitunog r^e^pon^iibiiiUes 
for.program and tmdget 



Targete^t^the nonpubltc^ctur 

.State's role m ^vernance, planriing and niointuruuj uf pn^fdrn 
and budget uncluar « * 



■ w 



Descriptioit of the Nonpublic Student 
Auxiliary Services Program 

The two state lawS which comprise tne funding packafge fo^" 
services to npnpublic students can be partitioned into the 



following subsections: 

4 



• 1. "Compensatory education** means 
preventive and remedial pro- 
grams in basic communication 
* and computational skills as 
set forth m the state 
administrative code. 



\ 



Auxiliary services for 
nonpublic students 




"Supportive services for ^ 
acquiring communication . ' 
proficiency in the English 
language for chilaren of 
limited Fnglish-speaking 
ability" means programs' in 
English as a second language. ' 

"Supplementary instruction" 
means instruction provided 
for a pupil clas^ifi^d pursu- 
ant to state law as handi- 
capped; it_ is given in 
addition to the regul^ 
instructional, program of such 
a pupil, as set forth io the 
state administrativef code. 

*Home instruction" means < 
individual instruction given 
in lieu of regular classroom 
instruction €o a pupil who is 
unable to a<ttend school 
because of illness or injury, 
as set forth in the state 
admin istr at iHfe code. 



General services for 
nonpublic students 



5. , Examination and classification 
of potentially handicapped 

♦ pupilS' (i,e., child study team 
services) • 

6. Corrective speech services - 
(articulation disorders) • 



order to focus the discussionr oftly evaluation problems 
dealing with Category 1 (compensatory education) will.be dealt 
with m the subsequpnt disgifssion. 

Services falling under "co^ensatofy " corresp6nd to the 
generally accepted definition of a preventive and remedial basic 
skills programs as^rovided under the state law^. Therefore, only 
services which are "compensatory" would be subsumed under the 
program evaluation procedures (TIERS) currently being implemented 
statewide. , 

The one ma^or • problem with program evaluation under the^state 
law governing nonpublic education is the lack of reference to 
program evaluation requirements in^either the law itself or in 
the interpretive and guideline materials prepared by the State , 
I^partment : 



\ 

'Problem with lack of 
reference to program 
evalua^on in law ^ 



"At the close of the' school 
year, the district board of 
education shall submit to the 
Commissioner a report describ- 
ing the classification and 
corrective services provided 
by the district board of edu- 
cation purusant to state' law. 
The report shall be completed 
in a manner prescribed by t^e 
Commissioner and shall in-/ 
elude, but not be limited /to, 
such Information as the 
classif icajtion and corrective 
service provided, numbers of 
nonpublic school pupils 
served, frequency and/or 
amount of the service, and 
facilities utilized." 



There are several problems or concerns raised by provisions 

in the state law governing nonpublic education concerning the 

impleioentation and managen^nt of services: ^ 

* Services must be provided in 
a pon-dectar iail facility 
(i.e., students much receive 
/ services away from their usual 
environment) . 
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Problems with 4^P^^^^nta- 



^ * Services must be arranged for 
and managecJ by the public 
^^^^ School and .may not include 

" use of any staff employed by| 
the nonpublic facility. 



tion and management of Services may be arranged 

services ^ . ' either through contracting, 



hiring of staff by the public 
school, or through coopera- 
tives among more than one 
public school district. 



. . Services may be delivered in a variety of ways, 

depending upon such factors as number of pupils, kinds of 

services, location -of facMiti^s, personnel available, Ic^istics, 

funds available! etc. Some. of these ways include the following: 

^ ^* Districts themselves providing 

. * services to all eligible 

, - ^ * pupils for whom these dis- 

^ tricts are responsible; 

•*->/- 

* -Two or more districts cooper- 
f ating to provide services to 

I ' all eligible pupils attending 

^ • * , nonpublic' schools located 

with^in their r^pective dis- 

* ' # trict^, whether or not the 

X .pupils actually reside in the 

\ same (Jistrict where the non- 

\ ^ public school they attend is 

^ r located; 

Delivery Strategies * Districts provi4J.n^ services 

through a cdUnty Educational (| 
services commission; 

* Districts contracting with an 
educational improvement center 

^ ' ^ . to provide services; and 

* Districts contracting with a 
non-sectarian private school 

■ ^ to provide services" 

Each of t^ ^bove repiTesenta any number of variables which 
may-have an impact on the services. With the variations that can 
occur, assum^^tions regarding consistency in treatment conditions 
fall apart. Measureiiient oS program impact could be aggregated 
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only if reasonable", controls are buidt m to' retrieve relevant 
information on setting, types of services, *etc. Given the fact 
that services must occur# outside of the nonpublic school, the 
potential for a breaKdown in conunun^cation between the regular 
/• teachers and tfie oompehsatory staff increases.^ , 

Counter to the provisions under the state law governing 
nonpublic education are the provisions for nonpublic services 
under P^L. S9-10 <as amended). The ESEA Title I program, which 
requiring management by the public school, 'fosters much more 
consistent^ services m that: 

\ *!• Services are provided on-site in the nonpublic 
facility. 

2. Services are provided by st^ff specifically 

i^3entified as instructional personnel for the 
^ nonpublic facilities (since under 192, these 

personnel are -not employees of the nonpublic 
school). Services , axe provided through coordination 
with the administrators and staff in the nonpublic . 
school. 

TWe Title I program also represents a joint planning effort 
between public and ndWpubiic sch6ols in that a single program 
'Plan*is developed, needs assessments^ coordinated, and program 
evaluation procedures are designed for both the public and 
nonpublic components (while, these may differ in terms of 
* specif ics# provision for evaluation is present for both). 

^Another general pjrobl^m. with the state law governing 
^ nonpublic education is limited funds available. ^By the time ' 
^suitable faqfllitie's are formed, transportation is arranged, etc., 
the number of students who could be effectively served may be 
very small in some cases. The nonpublic monies are used to pay 
all of these costs. Cootrols on the size, scope, and quality of 
* services appe^'lr to be more limited than under Title I. 

The fact that services must occur "off-site" precludes any ^ 
provision for Comparison groups. This means that the only 
, reasonable model that might be appropriate for evaluating 

compensatory services under the nonpublic funding category would 
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be" tlERS Al (TIERS A2 'would not^ be appropriate unless, a large N 
were being serve<t and three test aaministr ators coul'd be 
•scheduled)* If 'TIERS Al is required, then issues pertaining to 
^nflicts with on-gomg testing m either the public or nonpublic 
schools arise* The question or consistency of services at 
various sites, etc., all enter into the picture* 

J 

Evaluation Questions, Evaluation Problem Areas, 
and Evaluation Needs 

1. Re-write (impact) on legislation to provide for a clear' 
^^luation mandate for those programs governed under the 
state law for ryonpublic education. 

2* Design a comprehensive state evaluation plan for the 

Nonpublic Student Auiciliary Services Program including: ^ 

* How are the appropriated funds spent by each 
categorical area? 

* How much instructional time does each student 
receive in each area? 

* What are the most frequently 'used models for the 
delivery of these services? 

* Which models are^ the most cost-effective in terms 
of , their operations delivery^f services and in 
terms of student impact? 

Y 

3. Develop a concept" paper on the state governance role over the 
nonpublic sector. 

4* Re^ne the Title I Evaluation arfd Reporting 'System (TIERS) to 
accotDOdate those special nMds and problem ao^as defined in 
the previous section. ^ 

♦ . * • 

5. Develop a comprehensive progradf and budget evaluation, 

reporting, auditing, and mbni tor ing * system. 

6. '^Develop an evaluatfon tra4ning • plan^ for staff in the ; 

nonpublic sector. 



\ 
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Problem Case Description No. 2 
Evaluation in the State of Sterner* 



Within ^Sterner State Department of Education, evaluation comes 
•under the dffice-of Planning and Evaluation. • ^According to the 
coordinator of the office, the office is fortunate to be staffed 
with highiy qualified personnel. The office staff is made up of 
three individuals, each with a terminal degree in the field of 
education, and two highly skilled secretaries. According to the 
coordmatot, methodological problems that might be prevalent 
among other state education agencies dp not seem to be a concern 
m Stfeiner. The' coordinator feels that the expertise of the 
staff withm the Office of Planning and Evaluation negates 
methodological problem areas. 

The biggest problem area confronting the Office of Planning 
and Evaluation is the problem of acquiring appropriate funding 
sources to effect proposed studies. There have been many 
examples where study proposals have not been funded. 

Management within the Steiner State Department of Education 
has been very supportive of the Office of Planning and Evaluation. 
The typical routine followed by this office is to prepare a 
formal proposal, obtain proper authorization before proceeding, 
and seek an appropriate funding source. Almost always the 
appropriate funding source becomes the obstacle that hinders 
implementation. Attached is a^rief synopsis of the funding 
problem prepared by the coordinator for one of the state 
legislators; 

The Problem 

Coroparei? with other states, Steiner seems to be the recipient 
of a^disproportionate shaije of federally allocated funds which 
are administered through^ducitional pro]4cts sponsored by the 
National Institute of Education (NIE) through research grants. 

The Cause i 

Projects funded through NIE usually ^result from written 

'proposals which emerge successfully from a screening process. 

Policies of NIE stipuj^ate competition among potential recip/ents 

fpr the available funds. The very fact that- competitiveness 

« * 
exists works disfavorably for Steiner. For example, some state . 

education agencies have on their staffs trained proposal 

writers. In -Steiner, the responsibility for writing a proposal * 



*A fictitious name for a state. 
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is usually assigned to a person in the office who most nearly 
relates to thf topic under consideration. That person preisumably 
'already has full-time responsibilities and will likely give only 
token/e'f fq^t to the task of writing a proposal, especially wf)en 
be reiilires that should the proposal be funded he will be given 
the responsibility of administering the project in addition to 
, his regular duties. Obviously, this procedure lessens t^e 

prospects of obtaining an ultimately funded proposal whictr'has 
^ bien subjected to the rigors of competition. If Steiner could 
afford t;he luxury of employing a proposal writer, the salary that 
. ' that individual could 'expect would be far be^ow salaries of 
^ comparable positions in other states. Agam^ competition would 
likely rule out the possibility of a project being' funded for 
Sterner. Some exceptions do exist.' However, when one compares 
the total anount of money received by st^^tes through educational 
research grants, he will find that Steiner does not compare^ 
favorably, with those states that ftave more wealth and more 
skilled manpower for developing technical proposals. 

* The Cure • 

* 

If proposed projects aj^pear necessary or useful for disbur'smq 
funds, the money could be allocated to states on the basis of 
formulas which address such "'factor* as school erirqllraent, popula- 
tiOHf per capita income, etc. Obviously, ^this procedure would * 
^negat* the necessity of competitipn for funds for educational 
research projects. This does not^mean# howeve% that competition 
*for grants should be totally eliminated. Certainly, by the 
' nature of soi&e proposed pro^iects competition is desirable. 
^ Discretion is necessary. 
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problem Case Description #3 
The Evaluation Needs of Differing AUalences 

Two and a half years ago, a state aepartment of education 
instituted a full-time position for an interne^l program ^ 
evaluation consultant. This position was designed as an 
alternative "to hiring independent {i.e.v external) contractors to*^ 

X 

perform evaluations requirea as part of federally or state funded 
projects. . . 

Since then, the demand for these services has more than 
doubled. The evaluation staff is now grappling with issues 
related to increasing its effectiveness withm the organization. 

In contrast with other state education agencies, this 
Department is comparatively small and nas limited regulatory 
responsibilities. .It is compr ised "^of approximately 125 ^ 
professiojial staff whp serve the State Board of Education.* Its 
ma^or function is to provide leadership to the state's public 
school system. 

The State Board of Education has five members who are the 
elected regiresentatives of the state's five congressional 
districts. The' Board'sJresponsibilities are-'to distri^iute state 
and^ feder'ally apportioned funds- to the sci)Ools, submit 
recommendations on education improvements to the Gpvecnpr an^ 
General Assembly, and to appraise the work of the Commissioner of 
Education (whom the 'Board .appoints) , the Department of Education, 
and the state's pubJic school system. 

Because the sftate strongly adheres to principles of local 
cpntroi and autonomy, the Department staff's priaary functions 
are to provide leadership arflS^slfc/fical assistance to school 
districts and to administer federal and state categorical 
educational prograuas. These 'services are organized in four 
offices of Ebe Department, each 'headed by an assistant ^ 
commtMi^er, and the office of the. Commissioner . The 
Coooaijsioner^of Education and Assistant Commissioners form the 
Department '8^ Executive Committee. Within the five offices are 
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thirty un^ts, each specialized to provide either program serv4;j^es 
^ to the schoois or support services to .the state Department of 
Education* 

The Planning and Evaluation Unit has the prime responsibility 
to assist the Commissioner in preparation of the Department's 
budget. A modified Program Planning and Budget System (PPBS) 
called Planning and Man'agement System (PAMS) was developed by the 
Department to coordinate the budget process. PAMS includes a 
self-evaluation component for each prograiE operated by the 
Department. 

Th^^'self-evaluation component is glared toward the 
information needs of the state legislature and (Joes not provide 
the level of aetail neeaed by managers of categorical projects. 

Prior to the availability of internal evaluation consultants, 
evaluation needS were met m two ways. Program evaluations were 
conducted by the prcrject staff or through contracts with 
independent consultants. In the first sii^uation, staff ^pically 
Jacked the ejcpertise and the time to conduct more than ^ 
perfunctory prdgram reviews. Contracts with outside consultants, 
'however, posed* additional problems. Th^ Executive Committee 
became disillusioned with independent contractors because their 
i;e8ults too frequently were characterized by one or more of the 
following concerns: 

• Lack of timeliness; 

• Lack of formative evaluation; 

/ • Overly biased in a positive direction; , 
Overly technical 

• bJot responsive to the informational needs of the 
Executive Cooimlttee and State Board; and 

• Lacic of st^f comn^xtment to the evaluation results. 

fo restedy these conternSr t^ Depa^ftment agreed to expand the 
evaluation role of the Planning and Evaluation Unit by adding 
staff with specialized evaluation . training.' Funding for the 
evaluation -staff did not follow Sdriven\» prescription that 
evaluation" funding not coae through the program budget. Rather, 
the line-Item entry of evaluation in federi^y and State funded* 
program budgets made internal evaluations possible. 

f 
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Fqnds earmarked for evaluation .are noted by th^ Plafinmg and 
Evaluati9^ staff auring^ the' internal proposal review process. 
'Pto]ect managers are contacted and informed of the evaluation' \ 



I 

services available through the Unit. If the project manager is 
interested, the evaluation staff mi'tidtes a process of d^fin^mg 
evaluation objectives. This process leads to a formal contract 
for ser^ces between the Planning and Evaluation Unit and the 
pro]*ect manager . ' , 

Ttte process, which is outlined m Figure A, is designed to 
facilitate responsiveness to each level of management m the 
State Department of Education and to overcome the coRcejrhs listed 
above . . • ^ / ' ^ ' ^ 

That the process is succes^tul is evidenced by increased 
demand for the service. The process has m4t its objective to 
increase the involvement of the Executive Committee in defining 
evaluation objectives. K\\ parties to the* evaiuatibn concur that 
-benefits come from the accessibility, communication, and gomroon 
understandings tliat the internal environment affords. 

'Because the evaluations ^Te conducted by management service 
staf|, not pro^Mro staff, the objectivity of the evaluators is 
enhanced. ^The evaluation staff is directly accountable to the 
Planning and Evaluation Unit director 4th6 acts as' a buffer and 
mediator to protect the evaluator's integrity. Proximity to -j. 
project staff fosters close* working relationships that increase 
the evaluator*s o^nderstanding of the ^J^foject staff *s View of 
their program's pur^o'ses, goals, and^rpblems* 

However, the evaluation staff frequently feels torn between 
the needs of project managers and higher level managers i^ the 
Department. ' / 

Becapse funds available for any given study generally are 
limited, and the studies must meet federal or state program 

requirements f<^ evaluations, the scope and 'depth of the studies 

< 

has necessarily been restricted. While program managers tend to 
•give high priority to' formative evaluation ob;jectives, the State 
Bo^rd, the' Comioissipper , and the Assistant Coptraissioners have 

/ t 




greater nee*d for answers ^to summative evaluation questions. 
Federal and state requirements typically encompass both 
categories of evaluation ob3ectives but funds are not adequate to 
provide thorough responses to either category. The evaluation 



staff efforts become fragmented in trying to respond t9^all of 
the information demands • 

Therefore, t^e Planning and Evaluation Unit is seeking new* 
ways to 'increase the efficiency and effectiveness of the 
evaluation, process." Some alternatives being considered are; 

■ST 
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FIGURE^ A 

Evaluation. Process 
Planning and Evaluation 



y The following information represents the basic steps to be 
followed by consultants in the Planning and Evaluation 
Unit xn conducting evaluation studies: 



\ 



I* Establishment of Evaluation Objectives - Cumulative 

:ies Evaluation 



^teps (In writing) _ i 

S^. Proposal or Plan Review*- Ser: 



Objectives (as appropriate) 

B. Project .Manager - Verification and Development 
of Additional Ctojectives (as appropriate) 

C. Unit Director - Verification anid Development of 
-'-'^ Additional Object>ives (as appropriate) 

D. Commissioner - Verification and Development of 
Additional Objectives (as appropriate) 

r 

II. Development of Evaluation Design and Timeline 

A'' 

III* • Development of a Contract Agreement with Project- 
Ma pager 

IV. Iropleipen^tation of the Evaluation Study 

y. Conduct a Monthly fivaluation Pjrograrrf Review with 
Project Manager and Unit Director 



VI. Development of a Draft Report 

• • Verification- & 

VII. Developroynt oj a Final Repott Approval^y 

' Decision Maker 

VIII*^- Development and Presentation of Guidelines 
an abstract of the Final Report 
to: 

* V ' , 

A. Executive Committee 

B. State Board of Education 



\ 
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m Soliciting independent/funding, forifetudies of broader 
issues raised by the state Board, the Commissioner, 
and the Executive Committee; 

• 

• Increasing the technical assistance given by the 
evaluators to program staff so that, pr'ogram staff 

; would assume great|r responsibility for conducting 
formative Goraponents of the evaluations; 

t V - • 

• Prioritizing requrests for- different kinds of " 

evaluation information? . , 

• Increasing the*uSe of clerical 'and Student intern- 
assistance for routi^ evaluation procedures; and 

• Increasing coordination of evaluation data 
requirements 'with the Accounting Unit and the School, 
Finance and Data Services informatiotl S)f3tems. 

'-None of these alternatives would be eaay to accomplish 
because fund ciits are occurring at all levels of educa'tioh and 
present staff workloads arl| already overburdened. 



Questions 



1. What other aiternatives couLi the Unit take to 
improve the respQnsiveness of the evaluation studies? 

2. What communication systems are necessary to mediate 
^ ^hAi^dif fering information needs of the 

(trganization' s hierarptrf?-^-^ ^ 

3», What are the major advantage's of 

a) internal evaluations? 

b) independent evaluations? v 

4.." What are the appropriate kinds of objectives for 
evaluations done by | . 

* ay project staff? 

b) management staff? 

c) independent contractors? 
5. What actions should the Unit^ t^ke? Why? 
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Problem Ca8% Description #4 
Tfe^ State Agency and Third-Party Status 

. There are times ^when State Efduca^ion Agency (SEA) evaluators 
have difficulty keeping their'^thircj-party 'status. . George, a, 
ficti^tious name for a person 4^ho was once my boss's boss, asked 

evaluate a jitogram. that he had helped, institute. He had an 
active interest in the success of the program. , The results of my 
evaluation were negative; the program was canceled. My report, 
n^ilurally, was not the only r^asoA for t^f cancellation. 

For this episod^T^tKe program being evaluated is* not greatly 
impprtant, but it shou^* be briefly des^cribed. The program was 
established to cre;3te cooperatior^ between the SEA and some of the 
state universities. Personnel from the SEA were given releafse 
time {at full pay) to attend classes, work toward advanced 
degrees, and provide services teethe Univ^sities (i.e., a 
graduate assistant) . The universities were to make: 
resources — professional time — available to the SEA cpr no 

The professors were given release time ^ provide thes 
^^^^ o«n.Yice8 for the SEA. w / 

The majot^^litason for the lack of'^success of the program 
relates to the ose of the university resources by managers within 
the SEA. Most; agency managers knew of the program, but di4 not 
know how to go about obtaining the free services offeredl (This, 
is, a gross over-siapUf Ication,* but adequate for our purposes 
here. ) • * 

It is important now to understand the organization of the 
:^valuat4on unit withiiyfthe SEA. I was an evaluator, reporting to 
the director or evalftfation. There were other units with *si(irklar 
functions in 'the department.' George was the aAninistrator of the 
department. Because of this prganizat^ional structure of the SEA, 
I was reporting to, and (in a sense) evaluating the same person. 

That was a very unusual ^uation in the SEA. Normally, our 
evaluations werc^ conducted oil fedeifal programs operated through 
the SBA, such as Title I, Title IV, Special Education, 
Handi^iappttd, etc. George, and other people in his department, * 
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were r^ot involved in th^ administration of any of the prog-rams we 
normally evaluated. Since George reported, to the Chief State 
School Officer, our unit (and entire department) was similar to a 
staff position as opposed ti^a line positign. i^were in essence 
jthird-party , disinterested evaluators housed Hfroin the SEA. To 
evaluate a program that George partially administered was, as 
stated before, very uhusual. The^ evaluation was a special c^se, 
for a short time period, and for one' report only. 

sVmi^though this evallfttion was a special case, I still had 
difficulty keeping a third-party mental frame. of reference. The 
difficulty, I must hasten to point out', was caused not by George, 
but by me. As far as I know, he had no problem with my status. 

The major reason for my concern was my newness in the 
position I held. at the time. I did not know George very well; we 
had then recently started working together. .1 was fairly new 
wit^the agency also. My lack of experience and knowledge about 
George and the agency caused s^ome insecurity on my part. George 
did not seem to be a vindictive perdon, but I did not know how he 
would react to .a potentially threatening evaluation report. As* 
it turned- out, he Was indeed not vindictive, and reacted very 
well. 

In addition 't(Lmy personal difficulties, there can be some 
professional problems with this type of evaluation. An 
evaluation lacking th\rd-party status is a no-win situation. If 
'the final report is positive, critics of the program can eaaily 
claim the evaluator is trying to hide something. The repprt may 
«have little credibility even with supporters of the program being 
evaluated. I£ the final report is negative, the evaluaTtor may be 
in a less comfortable position than I was. Superiors can be 
threatened to a greater extent than George appeared to be. There^ 
could be charges of disloyality, or trying to do a "hatchet job" 
from within. Future reports and findings may not be accepted 
well, because of the<;p^t associations betwee^i evaluator and 
organizational administrator. All th-ia reflects pn the entire 
evaluation unit, no^ just^he single evaluator' within the <jnit. 
* 
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She ciedibility. of the unit can suffer, which hampers other vork 
that is truly third-party.,^. 

The results Of this evaluation were not nearly as bleak as 
they could have been. George recomroended appropriate action be 
t^ken with regard to the program. There was no ajj^imosity between 
us after the report was completed. 
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Problem Case Description »5 
The Problem With Positives 

It doesn't happen^ften, but a, evaluation^ reports are 
entirely positive; there are no negative findings.^ The program 
smells like a rose, .with no thorns attached. Non-evaluators may 
have difficulty imagirvlng the questions that go through our minds 
in such a- situation. Wh^t dicj; I mtss? Have I been co-opted by 
the program's xdeal^^r personnel? Did I — intentionally or 
not — do a whitewash on the program? Should ll^^do^one more chi 
square, one more interview? 

It is possible for programs to meet all the standards agreed 
on before the evaluation. * While it may, be true that other 
standards could be applied (or the , standards themselves could be 
evaluate^^ that may involve adding new ^ules ^o the game. Also, 
sojoe clrents want specific information. Even then, ^ear Abby, 
why dp J have such a difficult time living with no negatives? 

Evaluators a^re tempted to search for .negative results. This 
is a tendency we share w^th- auditors and otAer third-party 
investigators. s I would like to discuss; some reasons why we are 
tempted to 'find negative results about the programs we evaluate. 

First', finding negative aspects about programs validates our 

existence and service. We are especially validated if we find 

soMthing^that. everyone else missed. Without our service, how 

would a'nyone know the flavs in a program? We can best see this 

idea 'of validation by defining the purpose of evaluation as 

assistance to decision makers. In order to *help* someone, we 

must start ifitb a problem. If we have no problem, we cannot 

help, therefore the money epent on evaluation has been Wasted. 

If 

If we cani)i&jt t^ll a decision maker that sodething needs changing, 

«^ 

we Have no service to oj^fer. Our jobs then have no reason for 
existence. <Spod grief, we could even be eliminated. After all, 
theire are isortages to pay, children to feed, and a ciit that has 
not l)unted anything more^darfgerous than a paper w^d. 




Another reason we are tempted to search foi:' negative findings 
relates to our view of our role, ''it^is easy for an evaluator to 
get carried awty with the image of an investigative reporter* 



striving valiantly to unccWir goyer\Knental waste a-nd 
(nef flciency. .We have, after all, aVraoral obligation to the 
publ'ic, and especially the funding sources. Some ^valuators do 
not view their role as similar to investigative reporters. There 
st^ll remains the image of the disinterested truth seeker. We 
• simply cannot depend on program managers to point out problems 
with their programs. 

♦ 

Finally, it is easier to write about negative findings 'than 
to write about positiv^ findings. Positive findings tend to 
produce a "so what" feeling on the part of the evaluator. When 
we put something negative down on paper, we need a lot of 
ammunition. We expect disagreement from program personnel, so we 
have to make'jin extra strong case. In order to justify' ogr 
findi-ngs, we must gather ali the support px^ssible from the data. 
This justif icatioo. even includes many of those techniques we 
learned m graduate school, suph as how to use statistical terms ^ 
so no onfe understands what we are saying. This takes up space in 
our reports *and we are aware that the value of a report is' 
directly correlated with its thickness. (Well, even if we 
disagr-ee, our boss Relieves that, so we have to turn out thick 
reports. Ever heard of a person gating a raise because of a 
thin report?) ' 

These are some of the reasons we are tempted to search for 
negative results. The temptations are powerful. We feel guilty 
if, we do not come up with at least one bad thing about a 
program. I think, however, we should fight the seduction of 
comfort in negative finaings and not be ashamed to present 
positive findings. 

Some evaluatots are so ashamed of positive findings that they 
will present them in a reverse manner. ("There is no evidence 
that this program causes harm to students'' reading ability.") 
This is almost a double negative. There is a difference between 



saying we found "nothilig wrong" and saying we foui?d "something 

I 

right". 

Positive findings should be presented witirThe^ame intensity 
and fervor as negative 'findings. We still^j»d^t justify our 
findings, and gathjet.all possible suppo/t from the data. If 
evaluatiqn findings (either positive or negative) are to be 
accepted, the rigor of any study must be evident to the readers; 

Evaluators should realize that it may be helpful to point out 
the positive features of a progtara. Perhaps the program managers 
only get a feeling, of security, not specific suggestions of 
things to correct. Even so, other people are involved with the 
program and interested in the results. Evaluation reports ate 
read by people in funding sources, oversight groups, governing ox 
advisory committees, as well as the program managers,* superiors. 
While we often moan and complain about the lack of response to 
our reports, we really do not know how widely our w6rds a\e 
read. Let us accept for a moment that in. some cases the system 
works, and decision makers do indeed read our Mports. A 
positive report can be a help to decision nialters at many levels. 
An easy example^ to show this help is the case of two programs ^ 
competina for the s^me funds. 

It is important *for evaluators to understand that we are not 
trying to sell newspapers to a jaded public. We may borrow 
techniques fron the field of -investigative reporting; we^ may even 
'borrow a degree of ^epticism from that field. Our purposes, 
howver, are different. We are supposed to be discovering the 
nature and worth of a p^ipgram. We may borrow techniques from 
other fields, such as an adversial pourt situation, but again our 
purposes are different. Idealisa ni|ay be very helpful in some 
lituations, but. nympholepsy is not^uch value to an evaluator. 




will at this point seek forgiveness from all supporters of 
the union between investigative reporting and evaluation. My 
words Above show an admittedly less than perfect realization of 
the benefits of such^ a union. I am personally not opposed to 
discus^sions of similarities between the two fields.) 
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This IS not a call for evaluators to search for som^tbing ^ 
positive to say. It is definitely not a callr to try to collect 
data that will st^pw a program in a positive light. ^ I am simply 
stating that the evaluator should not avoid, presenting positive 
findings. Nor -should an evaluator feel any guilt or shame 
t)eo€bse of a .positive report. I started this document with the 
comment that entirely pos|tive repcrts do not happen often. 
Perhaps that is as it should be. Perhaps other evaluators, like 
me, have searched tpr something f^egative. What about you? - 
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Problen Case Description #6 
Cooperation from Schools m .Collecti\g Data • 

A State Department of Education Testing and Evaluation Unit<» 
conanitte^i itself' to participate m a fJational Longitudinal Study 
of high school sophomores and seniors. Since the National Study 
had insufficient schools in its study for the state to generalize 
about students in the state, the State bepar^iJtent^s Testing and 
Evaluation Unit decided to compliment the national study of 12 
schools by adding an additional 50 schools — thus creating a total 
state sample of 62 schools. 

The study design called for requesting a sophomore and senior 
roster from each- of the selected schools, randomly choosin<5 
students from ea'ch of t^ese class rosters, and inviting these > 
particular students to participate in a three-hour survey. Each 
student who participated would be a^Ked to complete three 
booklets: Identification pages; a questionnaire; an achievement 
test, 50 schools chosen represented a variety of ^ 

geographical locations, and a mixture of /rban, suburbani and 
rural. cooBwnities. ^^cause of* the small^number of schools in the 
study, it was particularly isit^ortant t/bat the mix of schools be ' 
maintained in order to make some generalizations about* students 
in the state. 

To insure school cooperation with th^ study, an agreement of 
co^sponsorship was^ establish^ between the Evaluation Unit and ^ 
the State's Aasociation of School Principals, This arrangement 
^ wa» intended to provide principals with some' iftiditional 
information about the study and additional incentive to 
coc^rate^ Some phone calls were made by association members tp 
principals^^m selected schools to informally encourage their 
support. 

To prepare fq^ contacting sct^ool districts and school 
principals, a ^wo-day training of evaluation staff was conducted 
by a member of 'the national office responsible for the study. 
During the training, staff reviewed forms, procedures/ survey 
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booklets, etc.r whi.cn\the national staff prepared, and some 
tnought was given d^s to how particular materials might be 
adjusted to suit the needs of the state's study. At the 
conclusion of the trainir>g, five two-person teams were formed and 

school assignments were made.^ TeAms were told that a letter had 

< \ 

already gone tfe di^strict superintendentSr--with a carbon of the 
letter Addressed to principals, and a lettar addressed directly , 
, to principals was* scheduled to be mailed, the following week. In-* 
addition, teams were informed that they wou^d receive a memo 
covering revised instructions of copies of revised materials 
with,in a few weeks, at which time it would be appropriate for 
them to begin phoning principals to 1) establish the principal's 
willingness to cooperate; 2) ^eitterate the nature of the study; 
3) receive the name t^f the school staff contact person who would 
handle details. Lastly, teams were told that the school surveys 
•needed to be concluded by May 1, wbich gave them a total .of three 
months for the effort. 

During the month following training, several staff spent 
their tike reviewing all the national materials, editing where 
necessary, making decisions concerning change^ m directions, and 
deciding bow many copies of materials would be printed. 
Additional staff time was spent developing a management 
information system so that the entire process could be monitored. 
H^roughout '^he first month of the three-month study, certain 

assumption^ we/e made about school district participation. Since 

/ 

letters explaining the study were mailed to district 
superintendents on January 30th and schpol principals on February 
7th, no response from them interpreted as a posit'ive 

^•response. Believirtg that adequate time had elapsed for 'school 
.districts to send niegative responses, no efforts wexe made to 
confirm district approval other than some contacts made by the 
Director of Evaluation to confirm the^ willingness of the staff of 

^HlMt laf gest school district in" the state to have seven of its 
twelve schools surveyed. The Director called the Assistant 
Super in tende/it and was informed that the distric/t office requJ.red 

/ 
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completion of a form "Application to Conduct Research and 
Experimental Studies in the Brisoane* Public Schools", and so 
this form -was completed and* promptly returned. During the third 
weeK of February , .the Director visited the school district office 
aftd received a verbal assurance by the' Assistant Superintendent 
that the study wouHi oe approvea. Five aays elapsed ana a 
written response arrived indicating that the district would not 
approve the stuay. This letter, received 28 days after the 
initial letter was sent to the District Superintendent explaining 
the stuay, \came as a complete surprise. By February 28, all 
materials W%re printed, 52 boxes containing- three" booklets for 
each of the 3,600 students had arri^d from the national office, 
and teams were preparing to schedule school visits. The 
magnitude of the problem of having the largest school district in 
the state withdraw from the study was summed up quickly by the 
Director — the loss^of that district would end the study. ' * 

During the next several days, steps were t^en by the 
Evaluation Unit Director to open up other possibilities to / 
salvage the study: 

1. The State Superintendent of Education assured the 
Director that he would be willing to write a letter^ 
to the District Superintendent asking for a review 
of the decision. 

2. Several school principals in the district were 
phoned to assess their willingness to participate tn 
the study. ^ 

J 

3. The Director received permissipn from the district's 
Assistant Superintendent and the Chairman of the 
principals* group to make a presentation abotjt the 
stii4y at a district's principals* meeting the 

» following week, in order to seek their approval to 

proceed. 

.4. A conanitment was maoe by the district's Assistant 
Super i/ftendent that he would stand by the decision 
of the principals. 



*A fictitious name. 
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As a result of the Director 's 'presentation to the principals, 
all affirmed their willingness to participate, even though two of 
the ^even principals expressed; concern that other school 
activities ,roight present sojne scheduling and administrative 
burdens. Following the meeting, the Director went to the school 
distribt offices, and relayed the principals' decision to the 
Assistant Superintendent, who gave his approval to proceed with 
the study. i 

Because Evaluation Units are not always so fortu/ate as to be' 
able -to turn a "no" into a "yes", hindsight affords us with the 
opportunity to have a clearer understanding of^what might have- 
been done to avoid the "critical incident" ^described. 

Several tbougnts surface: 



1. To assume that s^Ml districts feel obligated to 
cooperate ^witb r«|Bsts from State Department of 
Education Evaluation Units is a false assumption. 

2. , Letters which seek to involve a school district but 

do not specify a procedure for expressing a 
non-cooperative stance may Lead to aq unfounded 
sense of confidence in the Evaluation Unit about how 
many "for sures" there actually are. 

3. Responses to written communications take time and, 
therefore, a low level of resources should be 
expended (salaries, printing, secretarial, training, 
etc.) prior to commitments being solidified between 
Evaluation Units and school Districts. 

4. Verbal assurances do not replace the need for 
written assumrances in situations where key 
decisions control the outcome of "the entire effort. 
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Problem Case Description »7 
The Politics of Evaluations A Bilingual Case Study 

# 

The commitment of the State Legislature and the State 
Education Agency^ (SEA) to provide equal educational opportunity 
to students of limitW English language proficiency through 
bilingual education is reflected in Putlic Act 78-727. Enacted 
in September I 1973, PA 78-727 manda-fcecK the establishment of 



transitional bilingual programs in public schools effective July 
Ir 1976. Prior to this date bilingual programs were conducted by \ 
school districts on a voluntary basis. This Act enabled the 
5tate Office to provide supplemental financial assistance to 
local school districts to help them in meeting the costs of their 
bilingual programs. During that first year of 1976-77 13 million 
doj^lars (9,750,000 for Perth* and 3,250, 000 for all other school 
districts) were available. ^ 
Transitional bilingual programs are mandated ^n all 
^^^ttendance centers with 20 or more students of limited English 
language profidiency of the same language background. Districts 
with fewer than 20 students as specified may provide bilingual 
programs on a voluntary basis. Only the "transitional" local ^ 
* efforts are reimbursable by the State Education Agency. Local 
education agencies can choose to "go beyond" transitional efforts 
if they are willing to pay the additional cost or seek federal 
supplementary funding. 

Puring the tenure of the last State ^Superintendent, the SEA 
bilingual program administration ntoved from a one-man staff in 
the state capital to a' staff of fifteen professional and. support 
staff in Perth. The physical move north of 200 miles was 
sensible programmatically since the vast majority of the students 
and bilingual programs are in Perth or suburban collar %c\)ffs\%. 
However, this move permitted the bilingual staff to become 
isolated from many support sections^within the agency. 



*A fictitious name. ^ 
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The present State Superintendent established the Federal 
Prog/ams Coordination Council (FPCC) in January, 1^76, to develop 
a clearly defined, centralized intra-agency mechanism for 
• * examining the policy impact of federal programs on the total 

programmatic, ^administrative, and fiscal operations of the SEA. 
The Council, composed of the agency's assistant superintendents ' 
and several directorsr proN^ides a forum for the collective 
Dudgments of agency ^taff responsible for policy ' formulation on 
federal matters. Its recommendations are forwarded to the State 
Superintendent for review and consideration. The FPCC has 
i developed procedures to process federal requests ffi)r proposals, 
agency xesponses to proposed federal rules and regulations, 
requests for state endorsements of federal applications and 
issues. 

One of the procedures for quality control that was and still 
is utilized by the* FPCC is a^routing slip format. Once initial 
^ approval has been given by the Council for a ptogram section to 

pursue the submission o^ a proposal or state plan, final approval 
must be obtained from the Council after^a routing slip has been 
initialed by various sections within the agency. One section 
that reviews all proposals i^s the Program Evaluation and 
Assessment (PE6A) Section. 

For a period of one year it seemed that the Bilingual Section 
had tHe most serious problems with the Council in obtaining 
approval of its proposals .and especially its staffing ^iaM. 
V After a series of incidents, the directors of the Bilingual and 

PE«rA Sections reached ah acceptable arrangement in tfie hopes of 
modifying this situation. One of the staff in the PE6A Seqtion 
in the state capital would have her salary funded out of a 
combination of fecjeral bilingual plans; tl^is in<!itiduall would not 
only perform the required evaluation tasks but would also serve 
as a the state capital liaison to the Council. In this capacity 
^ the evaluation staff member could assist the Bilingual Section 

staff in drafting proposals and planning. ' 
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This resolution had occurred in the late fall of 1976. In' 
the early spring of ISpi , staff from tne PE&A Section were askea 
by the Executive DepiWy Superintendent to gather and review 
information for top level management m the agency about the 
management of the Bilingual Program. Fortunately, that staff 
member who reviewed the program had a head ftart in beginning to 
understand the program. ^ In the following months, the Bilingual 
Director left the agenCy and the appropriations bill.* to fund the 
entire state ^'program passed the State House by one vote. 

Concurrent jwith this litter series of events, the internal 
evaluation report produced' by the PE&A Section strongly 
recommended an immediate evaluation t^ determine whether or not 
the program was "trana-itioning" students from the bilingual 
pro9»am to all English c^l^iai^ooms. Yet to be determined was 
whether these students ^ere really learning English, or was this 
program a way to maihtai|n Spanish language and culture in 
American schools while dimuitaneously employing I*atino teachers. 
Not only was there no^data available, there was no system io ev 
produce data; this seemefd ironic since v/itho^ut the data system, 
program personnel would Inever be able to prove to legislators 
that the ^c^aram was reajlly having beneficial effects and was " 
accomplishing it^.. goals.' The internal evaluation report 
recommended that/at least an eighteen-month effort be initiated 
which would obtain pfeliminary findings and establish a system 
for future data collection. 

Howevern' ^the fijial latency decision to have a formal • 
evaluation of the Bilingual Program was -delayed beyond the point 
where the amount could be placed in the annual State Board of 
Education Budget Request durinc( the late fall of 1977. A 
decision was mad^ to introduce an amendment to this budget in May 
1978 in a State Board committee meeting, then to the full Board 
and then to the General A's«embly. In the meantime^ options were 
discussed internally with regard to the ambunt of money that 
would be necessary to conduct su<?h an evaluation and whether it 
should be done internally by temporary staff or by an external 
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t;hird party contractor. In either case the decision was made 
that- the Program Evaluation Section would bfe in charge of the 



effort and the difector of that section would' be the proje 
officer. Unfortunately throughout this period of discu$sb^n 
there'was no leaderlship in the program area — applicatijpt^fis^j^e 
" being reviewed for the va^«nt^director ' s position. I f 

The introduction of this fiscal c^pendment througjr the State 
Boara went ve^y smoothly .x/The general Assembly also accepted the 
amendments as a ^*quid pra gx^^^nr continujed funding'' of the 
^progr^^ In fact, one of th^ legislators known for^s 
resvvations about the proq^m introduced .th^amendment for a 
third party W^luation to exceed $13o\oOD for. FY 79. ^ 
The nextNfltep Wa^^^or the agency to issue an RFP iRequegt for 
Proposal), to asTarge a group, of bidders as possible. However, - 
firat the questions to be answered in the* proposal had to be \ 
^-determined. ,TlJfe- Bilingual Program director has been hired by 
thi^ time; he had seveiral questions. The State Board of 
'Education had questions; , so .did the SEA Planning Section that is 
re'sppnsible for developing an agency policy position on bilingual 
education, fh^ GeneraJ. Assembly h^d questions as well as top 
level SEA executives. \n addition, the Progra m Ev aluation 



personnel were told that it would be politicaJ^V prudent to 

discuss the RFP and the questions vith both the ^^erth Board of. 

Education (since 7$ percent of the students are io the Perth 

progjcamt^ as well as the Bilingual State Advisory Council, a group 

ilisi^ed by statute to set directions for biHngual* education. 

4 The discussions pointed out the differences among groups and 

amonq individuals within groups. Some viewed t^e effprt as 

evaluation for destruction; others viewed it as evaluation for 

^justification and others as evaluation simply to jdesctibe the 

facts as "they existed.' Many wanted their oyn questions included* 

in the RFp at the expense of the questions of the others. 

There was every attempt to expedite the development of the 
'' » 
RFP, and thfe review proces6. Although the original recommendation 

had been for an 18-npnth prefect, the s^tion was inforgied that 



practically and politically it would have to be completed dyring" 

the 1978-79 school year. Icjeally, it would be, done oy 

mid-December when the State Board was in its budget deliberations 

for FY 80. Of necessity, some information on the most important 

questions had to be available by the following May-June debate on 

bilingual education m the General Assembly. * 

After the contract was awarded, an Ad Hoc Evaluation Advisory 

was appointed that contained the following eclectic 

distribution: Senators^ and Representatives who are the major 

declared friends and foes of bilingual education; local aistrict 
i 

per3onnel from a rural downstate district, a suburban collar- 
district, and the City of Perth; university experts in bilingual 
education who tiad either a .qualitative or quantitative 
background, aftd the President and Vice-President of the State"^ 
Bilingual Advisory Council. This groujJ eventually met five time# 
during the eight-month period of the contract. The dialogue 

.among members of this ad hoc advisory panel and between these 
members and the contractor sensitized everyone about the 
complexity o*f the measurement issues. 

.During the first week of March 1979, when it becomes clear 

^that^hftue . would i>e a^^auocesefiil c omp letion to thls~ cohtraot, the 
cofttj/act project officer held a meeting with the Executive Deputy 
Superintendent to discuss the agency's approach to responding to 
the evaluation.* It was concluded that the agency response to 
this evaluation report would be at leaA as important as the 
report i£self to the General Assembly and their staff. Any hint 
of-defensiveness would be disastrous for program funding and 
perhaps even its very existence. Instead the' agency -^would'U.se 
the evaluation report as ^ springboard to begin making the 
program changes that everyone knew were needed. The questions 
did remain though as to h6w to orchestrate this response. 

It ^ was decided fhat^the Bilingual Program director could not 
chair the response task force. Since there was a possibility 
that there would be recommendations in the report. Jihat, as a ^ 
program mana9er he could live with but not as -an advocate for his 



own bilingual constitutency , the logical choice was *^onieoj?e at 
the Assistant Superintendent level who had three of his managers 
closely involved with the^ pro3ect. Therefore/ the Assistant 
Superintendent of Research, planning ^d Evaluation was named to 
chair this task force. ^ ^ 

As the deadline for the contractor's draft report approached, 
there was an attempt to clarify the role of the response task 
force. Two key players in forming this role were obviously the 
project officer and the^^rogram direfttor. A raeroo'was developed 
which outlined the sequential five steps necessary for the 
agency's strategic response to the evaluation. 

The steps in this memo proceeded routinely until step 
four — the presentation of the final draft to the evaluation 
advisory panel. During' that last advisory gi^nel meeting (June 
5) , several comments were made by SEA program staff -that could 
have been interpreted as attacking the credibility of the 
contractor, particularly on the collection of the transition 
data. The.tiline for these questions had be|n in the technical^ 
response to the contractor and not before members of the General 
Assembly and their staff. These remarks cduld be interpreted as 
meaning that there would be "btfsiness as usual" and that the 
evaltiatidn would have little impact. 

To avoid this erroneous conclusion, the program director and 
the project officer were instructed by the Executive Depu-ty that 
same day to begin drawing up the SEA response to the evaluation * 
even though the contractor had only delivered a draft report. . 
T^e focus for the response would be the contractors* 
recommendation section in the executive summary. The program 
director returned to Perth and^ompleted the se^ctions concerning 
program planning and policy; the directors of Research and 
Statistics, Program Evaluation and Assessnfient, and Office of Data 
Management developed responses to the recommendations of^a future 
evaluation plan within a general management information system. 

The backdrop for the imipediacy in these actions was twofold. 
First the House^had cut 8 millicin^^'^lar s^ from the SEA request 
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for Perth and 2,3 million for the downstate request. The Senate 
needed to know iimnediately how the SEA .was going to react to the 
external evaluation report so that Senate advocates would have 
something to point to as tS^ey lobbied for the restoration of the 
funds. Hou^e advocates also ne^ed something when pte time came 
for ttie Senate-House compromise over the appropriations. 

Second, there was to be a breakfast meeting for various 
•members of the House and Senate and their staff who were very 
concerned c^bout bilingual education. This breakfast would occur 
at 7:00 a.m. on Wednesday, June 13. Originally the subject of 
this seminar session ^as to be collective bargaining. However, 
the governor's office requested that a substitute issue be^ found 
and bilingual education became the topic. ANpanel of three 
memb^rrs of the General** Assembly would react to the evaluation 
findings. It became ' imperative that they also have the 
simultaneous opportunity to react to the SEA's proposed 
4ecoinmend\^ions or plans to implement the contractor's 
Recommendations. - p ^ 

On Jun^ S and 7 ttie- first drafts of the SEA reaction papers 
were written. The Projeqt Officer merged the sections on June 7 
and sent t^em to the Executive Deputy for review. On June 8 the 
comments for revision ""from the Executive Deputy and 
Su^^inten<Jent w^re* incorporated into la final draft which was 
approved ^^'^nday rao|ning, June 11. Copies of this reaction 
were then. )jai.f^^ to e^ry member of the General Assembly except 
those pane^l' members; tMeir copies were hand delivered. 

The Strategy worked. The breakfast participants were 
impressed with the speed and quality of the response. The final 
appropriation was 4 million dollars ft>F dovmstate programs (an 
increase ofMOO,000 from the previous year) and 12.6 million for 
Perth, antincreaie of 1;6 million) 



lie of 1;6 million) . 
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. Problem Case Description #8 

■ ■ ' . • 

Quality Control 

* > • . " ^ 

The' Education' Bureau is responsible for the collection of ' * 

data on student achievement for students who are in categorically 
aided programs^ Individual data are cdllected in the areas of 
reading, math, writing, and bilingual programs. Reports are then 
prepared by the Bureau in compliance with state and federal > P 

mandates. 

The Bureafu is faced with^ the problem of quality cor^pl in ^ 

ensuring that the data collected are accurfte and usable on the 

local, Aate, and federal levels. Before the data r^ch the 

state's Education %ureau, the data may have been handled by many 

persons. For example, the reporting forms may have been filled 

out by several different teachers, submitted to the principal, 

forwarded to the chief school officer, and then forwafSed to the 

superintendent of the Region who, in turn, submits several 

district's reports to a Regional Computer Center where the data 

a^ entered^n a magnetic tape to be sent tp the State Educatipn 

Department's Bureau of Evaluation. Problems arise when the data 

received by the Bureau are not machine processable. Since ^he 

Bureau uses a computer, the data received must be in an 

f*^ acceptable format. Schtel districts must complete several forms 

on each student 'and' the nuntf>er of forms and type of information 

requested vary* in accordance with the type program ^a student was 

enrolled in and the evaluation design the district chose for that 

program. In the pajst, the Evaluation Bureau has spent many 

montbs screening the da^^and corfecting such gross errors as the 

use of the wronq district code or an incorrect or missing car^ 

identified. These errors are the tip of the iceberg. Many 

ft 

errors, whiqh the Bureau cannot correct without cbntacting the 

districts for the correct information, skew thV^jresults. The ^ 
Bureau has developed a list Qf errors which repeatedly occur and 
has informed school districts th;at^they will be responsible for 
the correction of their data errors this year. 
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The following is a partial list' of types of errors which will 
be detected by the computer program: * 

a. Improper P9pulation code, e.g., ncmpublic school 
pupil assigned to a public school building. 

b. Improper ^oiopon^nt code, e,g., impossible code 
number. 

m 

r 

^ c. ^ Improper test code used for both norm- and 
criterion-referenced tests, 

d. Improper test level,- e.g., 2nd grade student given 
high school level test. 

e. Improper month for pre- or posttest, i.e., out of 
the normmg period. 

f*. Table missing. 

g. Raw test score missing for pre or posttest/ 

\ 

h. . Normal curve equivalent or Percentile missing for 

pre or posttest. 

i. Birthdate missing for pupil i-p'^n ungraded class, 
j. Duplicate data on s pupil. 

k. ^ Improper sub test code, e.g., vocabulary code used 
with matheutics test. 

Errors are classified into three types. Type 1 errors are 
critical errors and they must be corrected for any analysis of 
the data.^ Type 2 errors ^re considered substantial and they must 
be corrected for more meaningful analyses. Type 3 errors are 
classified as "information". Generally, they cannot be 
corrected? however, they may alert recipients to the need for a 
modification in testing procedures* or a change in the evaluation 
design ^n the next school year, A Type 1 Error would be a 
missing district code. A Type 2 Error would be when a math 
coaponent is listed for a pupil, but the number of contact hours 
in Math for that pupil is missing, and a Type 3 Error would 
indiq^te that the test administered may be too difficult for the 
student. 
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The Burea6 collects information on approximately one million 
pupils. The task of data collection is enormous for both the" ^ 
Evaluation Bureau and the lo^al. education agencies. Assistance 
is neeced m-the area of quality control and the development of a 
system whereby the data are screened along various check points 
will help to ensure the data received by the Bureau are as 
accurate as possible. The error correction procedure is an 
attempt «to begin this process. Additional elements need to be 
added to^jfhe errorlcorrection procedure to complete the quality 
control system. The implementation of the error correction 
procedure and other quality control elements are essential to the 
rapid processing of data that will make possib^^ the timely 
return of information to local education ag^cies, the state 
educatj^n agency, and the federal government. Without * imely 
return, the utility of evaluation for decision making purposes is 
lost and the evaluation simply meets reporting requirements. o 



41 



47 



FRir 



Introduction 



Problem Case Description #9 
Basic Skills Evaluation 



At the end of the last legislative sess-ion, a bill was passed 
which dealt with Basic Skills. One of the requirements of the 
legislation was to evaluate the state Basic Skills program. 
Funding for the program was for staff, sta^f expenses, and • 
ih-service costs. The evaluation budget was zero. 

This bill was negotiated at the "last minute" primarily by a 
few legislators and with minimal consultation "with the State 
Department. In a sense it^ was a very small token for the 
Governor who had wanted ahother very large education bill. 

General Background 

There were many logistical problems in getting this program* 
organized and off the ground. In the Department there were two 
~s«*^l»-af-pfriiosoptTy. one gtbu^ thought t>asrc"^skills were as 
» defined in the legislation, i.e., abilities to liJten, speak. 



consultants, -felt basic,' 
subject, i.e., science. 



skills were an integral part of every 



and thus they should be involved in the 

statewide planning. Tt^ issue was finally resolved and the 

L led . 
t \ 

A Department task fo^ce was developed with people from each 
Division of the Department to assure across-Department 
coordination and to guide program direction. A Basic Skills 
supervisor was hired along with 11 Basic Skills specialists. 
These specialists were assigned to/w^^^Jc in regions of the state 



in interMdiate unit offices* 

However^ the regional specialist reported to Department^ even 
though they were housed in and worked with staff of regional 
units. This was "upsetting" to many directors of regionalNjnits 
who felt if a person were housed in his office he/she should Kave 
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read, write and compute. Another -group, 'imposed of subject ar.ea ^ 



some adifrtnistrative responsTbility . Other state and federal 
programs had provided staff to intermediate units and all were 

r 

under the direct administrative control of the regional unit ^ 
director. 

^The operational model for the Basic SIcills program is the 
following: 

1- All Basic Skills specialists are trained 
&;imuj. t a n eo u 8 ly . 

2. After training, the specialists will then tram 
staff from districts who wish to participate in the 
p&ogram. 

3. After training^ the local school persons trained 
will then implejnent • basic skills program that 
reflects the Basic skills Standards of Excellence. 

.Aaother issue was the relucj:ance to make the standards too 
specific. There was a feeling that perhaps if the standards were 
kept ©ore general r accountability might be'^easier. 

Assignment of Evaluation Responsibility 

No evaluation funds were included m the legislation. Staff 
of the Evaluation Unit in the Department was assigned the 
evaluation responsibility as an{ additional task to its current 
activities. 

gyoqraa Implementation • 



The Basic SkiJ 
highly Successful 



ills program was modeled after a previously 
program "RigHt to Read". Erfcti ^gional person 
•would' train per^ns from local school districts to be the leader 
and coordinator of Basic Skills activities in their districts. 
Persons from\local schools wou^d be trained in both, content and 
process. From December to April, regional* specialists would 
explain the Basic Skills program to local schools and encourage 
them to pa^r^i^ipate. Prcan April to June there would be training 
of persons from the volunteer schools'. Training also would take 



place, again after the summer. The local schoqls would then begin 
to^^i^lement the Basic Skills program. 

The Basic Skills program had, a series of 18 statements, some 
measurable and some not, cilled "Istandards of Excellence*. These 
w^re the standards to be used which defined whiit an ideal 
comprehensive Basic Skills program would include. 



Preliminary Evaluation Task^ 

» • 

ProDlem I . To determine what the legislators had m mind as 
evaluation outcomes of the program. 

Action . Prepared a\pu:oposal that suggested that the 
principal authors of the legislation be densulted and shown some 
al^rnative questions which might b^^^^^asonably answered m the 
next year Uince a report had to be made to the legislature by 
^the following January, six roo^thB beJfore the funding and program 
adtivity would be completed aflW funding would^nd) . This would 
attempt to make decision makers moreC^nvolved m the evaluation 
process^lmd possibly encourage ttiem to utilize evaluation results 
for decision making. It would also clarify what kind of 
information was needed^at ti>eir level in ^c^trast to the kind of 
data tbat is neede<} for program%management ev^uation. This 
procedure was rejected by Departiftent decision mak.ers. As a 
result, an evaluation plan was prepared whicIT may or may not be 
of interest to legislators who will determine the continuation of 
the project. - • 

One ma^or issue well could be that only very preliminary 
information will be made available by January and that the 
legislators are expecting outcome data* 



Problei 2 , The Standards of Excellence were generally vague 
and not measurable. 

Action. After a series of meetings^with the Department 
COMittM and Basic Skills specialists, measurable criteria were 
"established for each Standards. It wis important that these be 
aeasurable so that data could be obtained from all participating 
^districts as base line data. 
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One raa^or issue m this area is that smqe there would be no 
new basic skills activities m schopl districts until four months 
before the preliminary report to^ the legislature was received, 
very few new activities probably would be impieinented . It is 
critical that the report clearly explain the meaning of base line 
data. 

Problem 3 . Can any new basic skills activities m schobl 
districts be directly attributed to the new Basic Skills program. 

Action, There are other state and federal programs relating 
to basic skills. There is no way to clearly establish a direct 
relationship between this new ^Jrogram and improvements in basic 
skills programs. 

Problem 4 . Was the training of local school persons 
effective? 

Action. All workshops needed to be evaluated to determine if 
th*e participants were learning the skills and processes, to ' 
implement a coordinated integrated basic skills program m their 
district. A workshop evaluation questionnaire was developed (see 
Exhibit 3) . 

h major issue was the concer^expr essed by the Basic Skills 
staff, many of whom had 'little experierlqe in putting on ^workshops 
or m having their "performance" ^^jaluated. An administrative 
decision was made, however, that ^the evaluation would be done. 

Problem S . How to measure the impact of the Bas^c Skills 
program on improved basic skill test scores. 

Action ,, It would be impossible to establish a measurable 
relationship between a regional training program and student 
achievement. Changes in basic skills scopes will be monitored^ 
through the State Assessment Program, over at least five years 
after the inception and hopeful continuation of the Basic Skills^^ 
program. 

Problem 6 . If no hard data are available how can it be 
determined if the . Bas W^Skills program appears to be making 
positive changes in school districts? 
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Action . A process evaluation plan was developedC^> An annual 
survey would be don^ on a yearly basis; the responses would be 
based upon the profession^ judgments of the respondents: lay 
ana advisory council persons, sdrhool board persons? schoO/1 ^ 
superintendents, and schpol staff. Their feelings would be the 
principal data which could be shared with decisi»on makers as an 
indication of program success* 

A raajc^ issue always is the use of judgment and opinion 
data. The precedent for this process had been established and 
accepted m the former Right to Read program. Because of the 
simj.lara,ty of the models of the two programs (Basic Skills and 
Right to Read) judgment data should be accepted. However, if the 
current positive climate in the state toward e^cation changes, 
the impact of judgment data could be reduced. In programs such 
as this vJhere the outcomes are not clearly defined and m which 
the content areas cuts across many other ongoing educational 
activities, it is difficult to gather, hard data. 

Problem 7 . Funding ^or evaluation activities. * 
Action . Miscellaneous evaluation expenses needed to be . 

covered by other program funds. Substantive evaluation is, 

therefore, very much constrained. 

Evaluation Summary 

An evaluation needed* to be done. For various reasons the 
Evaluation staff was unable to meet with the legislative decision 
makers. This was an example of the role of politics in ^ 
evaluation. Program evaluation at a state level often is a long 
way-away from the "textbook" evaluation procedures taught in 
, colleges and universities. Knowing that the evaluation that was 
to be done was going to be less effective than it could have begn - 
perhaps raises the issue of the professional integrity of the 
evaluator versus a need for survival in his/her job. 

Will the proposed evaluation activities have any influence on 
the final decision, for the continued funding of the Basic Skills 
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program? Probably some, but not a major factor. Only when 

evaluation is considered when legislation is developed ^ can 

♦ * \ ^^^^ 

evaluatipn. results have a chance of bemgXa maj^r factor in 



continued fundings of ft<fgram, 



am^ 
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Problem Case Description No. 10 
Monitoring - A Threat to Evaluation? 



ERIC 



Things started to go wrong on Jack's third visit to the 
program funded by his agency. He felt he had lost all ralpport 
and cooperation with the program people he was to help. Even 
visits to other funded programs were not well receipted. It was 
as if word was out to beware of Jack. ^ 

During his first visits Jack thought he was accepted as an 
evaluator . He had worked with the program director to show that 
evaluation, with its myriad of definitions- dependent upon^ the 
different schools of thought, may be viewed as the assessment of 
the value or worth of a program or activity. He felt that 
evaluation could be seen as a tertiary relation^ (x, y, 2) , where 
"x", as an evaluator, acts with a set 6f data "y" to determine 
whether a standard "z" is met. He knew that decisions ba^ed upon 
evaluation results should be made by the program director or 
administrators. 

Jack was also please?^* with his second visit as a nvonitor. He 
explained to the program director and others that monitoring may 
b« thought of as a procegs^ for ascertaining whether a program or 
activity was within various rules, regulations, mijiimum ^ 
standards, or agreed upon terms. Monitor ing^ also may be seen as 
a tertiary relation, (x, y,. z) , where "x", as a monitor, acts 
with a set of data "y" tio determine whether a prescribed precept 
"z" is met. Jack knew that decisions bailed upon monitoring 
results were made by the 'funding authorities and policy makers. 

Jack was well aware of the strain during the third visit when, 
the program director asked, "Fpr what reason are you here? Are 
you h«re to help us with our prc^gram or are you her^ to be a 
watchdog?" In tlie eyes 'of the program director. Jack, as an 
evaluator, suddenly became, a mohxtor. ^ * i 

In the cdse f(i Jack, the roles of an evaluator and a monitor 
became confusing. At one point, he was performing activities for 
purpo8H8,of evaluating tt)e program, while at another time, he was 
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collecting information or ^Jjs^trving for monitoring purposes. ^Was 
th'is sensed'^lack of rapport duel to his agency having the ||^me 
unit or 'personi| responsible for jboth evaluati9n or monitoring?' 
Although ijacK probably had oo problem with the role change 'since 
the" foe of a ntonitor and evaluator are different, the funding 
recipient had difficulty in ftitetpreting *the intent of the ' 
visits. In effect, a relazed atmosphere necessary for 
c6ininuni6ating 9f).ear information to' an evaluator wa% lost. 

Jane, an evaluator in .another agency, also ran' into problems 
with h^r visits and communications with funded project 
personnel.. She was attempting to assist in' setting up evaluation 
designs collecting formative information when asked by tfie \ 
project personnel, "How are you goirt^ to use the project / 
information?'' Jane believe<^ tiiat she had clarified earli'er how 
data were* to- be used in an evaluation process and was surprised 
* at the further line of questioning, "Will the information be used 
■ to asaijBt in the improvement of the program or will the 
infotroa^ion be ikied to cut Or reduce funding?" 

^ith further^invAstigation, Jane, ibund that the project 
received a /lonitoring instrument by her agency-. The project 
persc^nel felt tha^ many of the questions in ,the instrument were 
too^ similar to those being solicited ^in the evaluation. The 
^^*ttose or use of the ir*or©ation became so unclear that the 
project personnel were afraid -that some information may ^e 
mi^nterpreted and used against them. They were fc 
program could be in jeopardy if they answered questions in a 
' ^^^^ manner, especially when some adjustments were presently 
^ being made to alleviate identified 'problems. ^ 

As in Jackj^s situation, the atmoshpere necessary for 
communicating clear information was lo^. Jane had to spend a 
large amount «of time assuring that information she collected 
Would' be used at the agency level for assessing the value or 
wc^th of aerogram and for providing feedback to assist in the 
devel^m^nt of the program. She further assured that her data 



on may be 
fearfull^that 



• ^uld n9t b« u 
regulatfons/ 



sed for monitor ning compldance to various rules or 
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Problems between evaluation and monitoring ^can occur whenever 
varipus corresponding components of the two tertiary relatipns 
are eqii^^^o^ elements within the components intersect. Jack's 
case illustrated a situation in which first components were 
equal. In this case,^the evaluator an^ the 'monitor were the same 
person^ Jane's case of having collectea some evaluation data 
similar to that collected on a monitoring instrument Exemplifies 
a situation whenever elements .within the components of the 
relations intersect (elements of the second component 
intersect) . This situatiorf can also occur whenever the same unit 
within an agency is responsible for both the evaluation and the 
•monitoring of programs (elements of the^first components 
.intersect) . Although it is theoretically possible for other 
component equality or intersection of elements, this author has 
not observed such instances. ThJTs absence may be a good sign as 
occurrences of other instance^ wolild indicate serious problems 
with understanding the differences J^etween evaluation and 
mo^i^ring. 

An ideal situation would be the case where no equality or 
intersection occurs. However, with organi2ational structures 
which place evaluation at a J.ow priX)ifity, with the limited 
knowledge base of various per^Hs about evaluation and 
monitoring, and 'in a time of staff reduction, -it is very' unlikely 
that the ideaL situation will occur in many funding agencies*. 
Thus, to lessen problems between evaluation and[H^nitor ing, 
strategies need to be developed for situations where 
corresponding components within the tertiary relatior^s are equal 
or where elements within the components of the relations ^ 
intersect. 

Strategies should bejplanned for addressing problems at both 
*the funding agency and the funded agency levels. Answers need to 
be found for such problematic questions asr How does one assure 
.to fui^d project personnel that evaluation results will be used 
to judge the worth of quality of programs? What does it mean to 
judge the worth or quality of a ptogram when..the question of 
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continued funding is in the minds of funded project personnel? 
What can be done to better delineate the roles of evaluators and 
monitors? What are some ways a ^funding agency can assist staff 
when the evaluator is also the monitor or is m the same program 
unit as is the monitor? What are the major differences in 
evaluation data and monitoring data, and how can these 
differenc-es be'^^est communicated to thos^ 'program policy makers 
or decision makers who are not evaluators? 

Although some answers to problems may seem quite evident .-to 
ones who are knowledgeable in evaluation ahd who, for example, 
can distinguish between evaluation and monitoring, one must keep 
in mind that the evaluator is but dfte actor out of roany who are 
involved with programs. As in Jack's case, the problem was not 
necessarily with Jack having to differentiate between the role of 
an evaluator and a rabnitor but rather, with the funded project 
staff feeling that a conflict of roles existed. Even though the 
funding agency may recognize a diffe^rence between an evaluator 
and a monitor, a solution to the problem by having different 
evaluators and monitors may not be that simple when funds are not 
available to hire additional qualified staff. Jane knew how to 
apply her data against evaluation standards, but the project ' 
staff was not convinced. Perhaps they felt her super los may 
misuse the evaluation results and make regulatory decisions or 
funding ^decisions. 

MoniAring — is it a threat to evaluation? Perhaps not, if 
possible problems are antic ipated^^and strategies develc^ped to 
address these prob'lems. 
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Problem Case Description No, 11 
Betwixt and Between 



^ Background 



7 



' For State' Of f ices of Education, accepting federal funds often 
carries ^ith it a responsiblity to gather evaluative data. The 
data then are reported to the appropriat^e federal agency.' 

Recently r however, in one area—Title I 89»>10 — federal 
aeta-«valuat'ion studies' concluded that "eVaiuation reports 
received f^om states were of doubtful quality and that no 
aggregable national data could be compiie^J^ecause of the 
diversity of the reports. The consequence was that in Title I 
'89-10 and other programs, federal evaluation guidelines now 
include the prescription of .acceptable evaluation models. It is 
argued that implementation of such evaluatic^i models as these 
will provide information which will allow sound generalizations 
to be made at the national* level. This may be true, but in the 
implementation of th^ models a situation, is created. where the 
State Office evaluator is caught betwixt and between providing a 
service basically for the benefit of a federal igency, providing 
evaluation services useful to State Office decision makers, or 
providing service? to Local Educational Agencies which has ^heir 
benefit as a major concern. ^ 

Nature of the Problem 

When providing evaluation for federal programs the major 
^evaluat^ion functions of the itate Office qf Education are often 
three-fold. One function is to provide service to Local 
Educational Agencies by providing ass istanc«^ which would-be 
conpatible with locail needs and which would add incrementally^to 
the current local level of sophistication. The second function, 
serving State Office administri^tors, includes gathering data 
regarding compliancy with federal law, gathering information 
which %#culd forecast 4>otential troublesome areas,, and gathering 
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data which would complement grant writing efforts. - The third 
function is to serve the "Feds**.* ♦his includes monitoring lEA 
-evaluation efforts, making the requioed annual reports, serving 
as an mte'f m^tltary between the Feds and the LEA; and providing 
technical assistance in 'the implementation of the federal 
evaluation models at the^ocal leVel. 

Of these three sets of functions, the primary concern of the 
State Office is to serve federal interests". That primacy derives 
from tire power of the purse. Because federal monitoring is a \ 
possibility, the emphasis withm the State Office is to serve \ 
federal functions at the expense of any other interests. 

At this point a variety of complications should be 
mentioned. Most of the complications reflect the limited amount 
of administrative funds allocated to the state. 

The constraint on funding has a variety of ramifications. . 
federal funding is administered m this State Office by progjp^ 
staff who are separate from •Evaluation staff. Program staff 
typically. argue that allocation of administrative funds needg to 
be directly supportive of local progr2un impact upon children. At 
- that point, program staff point out that Stata Office ^valuators 
primarily function as a service for federal interests. 
Evaluation, then, is assigned a priority by Program staff. 
^ Limited funding assures that only one or two staff are 
allocated to evaluative purposes at the state level. In addition 
y^o a limited number of personnel for evaluation, travel and 
additional resources are limited. 

The lack of staff provides a severe constraint on the scope * 
of work possible, with federal interests pr;,roary, state 
eva^uators often put other interests aside ^completely. In the 
eyes of the program administrator this confirms the low priority 
assigned to evaluation. For. the LEA, it virtually assures their 
frustration when anything but^ federal model evaluation is " 
broached with State Office evalultors. For LEA staff uninitiated 
to the scope 'Of evaluation-, it crea^^ p the impression that 
eicperimentalist and measurement oriented outcome evaluation is 
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what evaluation is all about. For small schools or schools 
lacking technical sophisticationr the models are viewed as 
byrdensome and without potential transferability to other 
evaluative purposes. 

Parameters for Resolution 

The quick and easy dasmissfel of tne "problero** as beirxg 

remediated by reapportionment of S^tate Office funds to provide 

for adaitional evaluation staff is not likely to b€ possible. As 

♦ 

was indicated, evaluation. is perceived to be a low priority for 
the persons administering the funds. Something must be done 
first by the State Office evaluAtors which will demonstrate the 
benefit of evaluation services. 

At the State Office level the demonstration should relate to 
such aspects of expected functions as forecasting potential 
trouble areas. This mus$,^e done with limited travel, limited 
Staff while at the same time the staff fulfills obligations to 
assist io the implementation of federal evaluation models. 

In aupport oL LEA evaluation efforts, considering many of the 
same points listed m the previous paragraph, what can the State 
Office evaluator do? Here thV-iindertakings must be flexible 
enough to adapt to a variety of LEA setmgs — including such 
features as size, technical sophistication, available lo<£:al 
staff, and restricted finances. 

Another quicK dismissal of the "problem" might be attempted 
by expanding the funding and fwnctipns of the Technical 
Assistance Center, idea as imp'lemented in Title I 89-10. But 
TAC*s are constrained ^by their basic function of supporting %hB 
implem|rntation of the federal models. Evaluation needs go beyond 
these nodels* The evaluation needs at the State Office level 
also would not -be typically' within the scope, of interest of the 
TAC's. Finally, the TAC's ofteti represent a hi^h technology 
approach to the solution of^ problems. High technology solutions 
are beyond many of those in need of evaluative assistance. 
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It may nearly be an impossible task to devise evaluative 
"* 1 ♦ 

tactics which can ^e circumscribed within such bounds. I 

believer however, that much m these parameters delineate a great 

neea for evaluation. We, as evai^ator^, -must have a kit bag 

which includes evaluation approaches or tactics which can be 

implemented by a few persons, m a short time, with limited time 

and funds? and m addition, many times to be implemented by 

persons or relatively modest evaluative sophistication. 

^ In our day with our fascination with high technology, it is 

easyl to forget the multituae of evaluative needs/ which will not 

be thu.s served. The forgotten needs will ^e those where money 

and power are lacking--yet they are not insignificant. It seems 

to me that it is there where much will be done to improve the 

condition of educat:^.on. 
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Problem Case Description No^ 12 
tHaximizmg Assessroent Results Utilization 

Probably one of the roost vexing problems existing for State 
Departments of Education are attempts to make assessment results 
meaningful at multiple levels of the educational hierarchy. 
Teachers want to use assessroent results for student diagnostic 
and perscriptive purposes, while building administrators look to 
assessment results for within building program and curriculum 
evaluation. District personnel perceive assessment results as a 
^ way of evaluating programs and curriculum between buildings, 

distr ictwide. At the state level, assessment results are viewed 
^in a broader context. Typical state utilization of ass^sment 
data are for evaluation, of students, programs and districts 
across the state. Often domain deficiency within content areas ^ 
are explored. The main utility of assessment resultfs, however, 
at the state level is for supporting evidence^ for additional 
' funding by legislatures or policy issues by State Boards of 
Education. Most statewide assessment programs focus on the 
latter uses of results, often overlooking teachers', building 
administrators' and. distr icts* needs for quick accurate 
assessment results for evaluation of students, cprriculum or 
programs at the local level. 

ParamouAt among the problems of multiple use of assessment 
results is the scoring of answer sheets. Scoring systems seldom 
offer feedback to local school districts or teacheraf, such that 
this infonuTtion can be u)sed for local decision making. Usually 
student responses to questions on statewi^de assessments are 
obtained under wcmt machine^ readable format, optical scan answer 
sheets being the most popular. The answers are collected by the 
teacher or test administrators » fend to the building level for 
aggregation of all classrooms assessed in that building, and then 
sent to the school district for further aggregation of 
districtvfde assessed classrooms and finally to the state tot 
-scoring of the answer sheets* In many cases states Send the 
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answer sheets' to a scoring service for processing. The results 
of the statewide assessment flows m reverse order in «report-ing 
the results back td the local school districts. This" scoring and 
reporting process/can take from several weeks to several months.- 
Therefore;, ij^^>4(e assessment- results aiy reported in such a * 
foriftat wirfcIT can be useful on the local level (often they, are 
not), they do not filter down to the appropriate level in a 
timely fashion. The irony of the statewide assessment scoring ^ 
process is that those that have the need for immediate test 
results, I.e., making stuaent decisions, often are the last to 
receive* the results, while those that are more interested m the 
overall picture of statewide assessment and ar^ urtable to act on 
the results unti^ next legislature or State Board convenes are 
the first to receive the results. 

The assessment reports which are generated from the answer 
sheets require different informatipn for each of the levels of 
utility. • There is the need. for very detailed and discrete 
inforfliation on the reports which are returned to the teacher m 
order ,for her/him to make judgments about students. These 
reports must indicate item by item how each student. performed so 
that the teacher may make judgments about students and design 
classroom activities which will facilitate student achievement. 
This report is ve'ry different from, say, the building level 
report, district report, or even the state reports which are^ more 
concerned with averaging of scores across domains^ reporting 
itejns for item analysis or rejSorting of gross scores across 
programs, buildings or districts. Usually as one rnoves up the 
educational hierarchy, assessment information is needed in a 
decreasing order of detail. 

Finally, there is always the problems of confidentiality as 

the reporting moves away from the classroom. If the assessment 

J 

data is to be collected such that teachers may use the results 
for classroom purposes students must be identified. At the state 
level there is little if any need to identify indivi^jual 
students, ho%#ever, certain student characteristics may be very 
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important (sex, race, handicapping conditions of students). 

While "reports that do not identify students, buildings or 

Classrooms may have little local utilit^, these identifiers are • 

usually not: needed for any statewide, assessment. State or 

federal confidentiality constraints or the degree of * 

confidentiality required of individual students raay impede 

assessment efforts at, the state, level m the future and therefore 

limit the utility of s^tatewide assessment reporting^ 

The utility of assessment results at the various educational 

levels neeas to be addressea by those doing the assessing, if for 

no other reason than to offer some guarantee that assessment 

results have a degree of validity. It is not enough anymore to 

expect local school districts, schools, and teachers to 

enthusiastically administer a statewide assessment because "they" 

need the data for "some" reason. Only yhen those administering 

the test instruments t«r^ludents can plainly see some direct 
k 

benefit can we hope for x:ompliance to standardized administrative 
methodology. The validity of statewide test results becomes 
particularly troublesome when teachers must administer a s^€ond 
instrument to obtain the data that will be useful for their 
decifsion making. 

A suggested solution to this problem is to have assessments 
initiated at the local school district, school or classroom level 
and then the results of each of these aggregated as they ar^ sent 
forward to the state. The problem with this approach is the l^ck 
of coordination in order to^ obtain comparable data to aggregate. 
There is a need for agreement on what is to be assessed/ how and 
when the assessment is to take place. Different testing times, 
nonstandardized assessment procedures and attempts in ag^egation 
of assessment results wher^.>ffttruments are not comparable axe 
major problems. 

At first t>lush the^above solution seems to have little or no 
merit. However/ if exanined for technical feasibility as well as 
practicality, it is quickly recognized tihat the current state of 
the art of scoring and the rethinking of out moded idea's can make 
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this solution a viable direction in which to pursue. The 
solution as posea, rests on theoretical'concepts which we have 
accepted over the years due to past technical limitations. 
Mainly, the^scoring of student tests must take place outside the 
school, or school district. This was due to the fact that 
roach merj to score stuaent tests was very expensive, -^ith the 
new computer technology, it is within the grasp of most schools 
and school aistricts to do their owr> scoring and report 
generating. With assistance from state and federal-governments 
assessment support materials could be purchased (computer 
hardware and software) that would produce a new type of 
assessmertt administration that would furnish appropriate 
assessment results at every level of the educational hierarchy. " 
This method woi^ld require that the student asse^ssment data be 
scored and ^epo?led at either the school or school district level 
and then data sent forward m a machine readable format, 
aggregated as it moves toward the state level. The 
instrumentation could be a joint effort by state and school 
districts of identifying areas to be assessed and' other 
specifics. This method of assessment would maximize the 
utilization c5f results. Those that need the results earliest *and 
in th'e greatest detail would have first call on the results, 
••while those that need less detail but a broadej>-d»ta base would 
also obtain results in a timely fashion. ^ ^ 
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Problem Case Description No, 13 - 
Coleroao Assessment Program 



Background 



^ Th«re has been a legislatively mandated state-testing program 
in Coleman since 1962-^3. The Coleman Assessment Program (CAP) 
'as It is currently krjown, tests all public school students 
annually in grades 1, 3, 6, and 12. 

The first grade test is an entry level testyadi^mistered at 
the^beginning of school to measure the readiness skills whic^ 
pupils bring with them to school. In the other grades^- 
state-developed testS'are administered on a matrix sampling basis 
to monitor the achievement in reading, language, and 
mathematics. No individual student scores or classroom^^reports 
are produced. The smallest unit of analysis is the school. 
School data are then aggregated to produce district reports and 
an annual report of statewide achievement. 

A state dir^tor and six consultants administer the program 
at the state level. Contracts are let" ^or printing, 
distribution, collectiohn, scoring, analysis, and preparation of 
/eports which are mailed to each of the 1,044 Coleman school 
districts. Regional workshops are held each fal^ to acquaint LEA 
personnel with the content and interpretation of the reports. 
All^assessMnt information is available to the public after it 
has been presented to the State Board of Education in November 
following the school year in question. 

School Vnci district reports include the raw scores for the 
current and prior school years. Look-up^tables provide norms in 
the form of state percentile ranks. Background factors, such as 
socioeconomic indexes apd English language fluency, are reported 
and usfd in multiple regression arialVsis to produce a predicted 
score. Reports also includ% student score distributions and 
subscores by skill area (e.g., capitalization and punctuation) 
and by subpopulaf ion >e.g*, boys vs* girls). 
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There are many issues, difficulties, dilemmas, which an 
assessment program faces. Few ot those can be resolved to the 
satisfaction of ail those affected by the program, but every one ' 
must be addressed. Each of the following paragraphs touches upon 
an issue with which CAP staff must deal. While the listing is 
not even exhaustive, it can 'nonetheless mislead the reader into 
Concluding that CAP has on^y problems. CAP has been notably 
successful. However the purpose of this paper is not to look at 
successes but rather to focus on problems. Hence the following 
focuses on the empty portion of the glass, not <he full portion. 

Problems ' ^ ^ , 

1» The dilemma of reconciling local autonomy in curriculum 

ir 

with a state program which assumes, or seeks, commonality of 
teaching objectives . Coleman does not have a prescribed state 
curriculum. Thus it is difficult to achieve consensus among 
educators as to what should be taught and thereby should be 
assessed. 

2. A large-scale assessgient program must rely almost solely 
on paper and pencil, machine-scorable , multiple-choice tests . 
While many t'feaching ob]ective3 lend themselves to such measures, 
many other skills — particularly in the minds of many 
educators — can be less directly assess in such a mode. 

3 . Shifting legislation and financial support make long-term 
planning tentative at best . Although CAP is legislatively 
mandated, those mandates have changed many times in 18 

years — which grade levels are assessed, what content areas are 
tested, switch from requiring IQ te|^ to prohibiting them. 
Likewise budget allocatipns are unkrtown unti^ the list minute, 
particularly in recent* years wi state initiatives such^ as 
Propositions 13 and 9. 

4. How to motivate students to put forth t^eir best effort . 
Because individual students have no stake m the results, there 
is little incentive for them to do well on the test^, 
particularly so with high school seniors. 
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5. How to distinguish between true instruction and coaching 
or cheating . While the state wants to encourage schools to teach 
the objectives of the test arid particularly to remediate skills 
previously found to be weak, sometimes these efforts could be 
viewe<3 as teaching the test .items tather than teaching the test 
objectives. \ 

6. ^ ^How td meet the'eonf licting gpals of cdmparative 
information and instructionally relevant information * Some 
audiences, such as legislators and the press^ demand normative 
(ranking) information for schools and districts. Those scRoolsi 
which rank relatively low become hostile and therefore aren't 
willing to look beyond the negative ranking^J;x)--discover in what 
skilll^iheir students are deficient. 

7. How to keep assessment reports simple and usable .without 
sacrificing completeness and accuracy . The recipients and users 
of the reports differ widely in \heir technical, statistical 
sophistication. ' It is difficult to^ reconcile the dilemma of 
keeping a report simple enough to be usable by the classroom 
teacher without offending the research staff of larger school 
districts who would label such a report misleading and simplistic 

8. How to encourag^e use .of* assessment information at the 

local school level . The complexity of the current reports (cited 

above in number 7) is only one factor inhibiting usage. Other 

factors are even more diffic^lt to, overcome — e.g., ant^athy of 

teachers to testing, the oppositioii to what is viewed as state 

control or state interference, feeling that teachers themselves 
* 

are being evaluated, frustration at repeatedly unsuccessful^ 
yef^forts to improve achievement in schools serving disadvantaged 
' students. 

9. Whether to keep tests constant to encourage longitudinal 
analyses or change tests to improve them . LEA ^Personnel I 
continually request the conflicting need to improve the tests yet 
keep them the same. How responsive should the state be to 
recommendations or suggest ioM?^^^>t is difficult to ascertain \ 
from isolated comments or committee recommendations hbw . 
representative those thoughts are. 
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— H) jHovr tcr deal with scores of students In special programs 
such as special education or bilingual . The advent of ^ f 

• mainstreaming for special education students an^ the increasing 
Jitiniber of ^students 'whose English^language fluency is limited have 
magnified the dilemm^ of whether to include their scores il5 those 
for the school. \V / 

11. Relation between assess^nt' and minimum competency A 
testing . Thus far these two programs have been viewed ^a^s two 
distinct programs; however, the layering on of additional^^'ba^ts, 
which results in duplication and triplication of testing of 
studehts (who a?fe in ESEA Title I) is aUL too obvious to laEAs and 
a method of consolidating' testing will be in demand \o save money 
and instructional time, / ^ ? r 

12. ^ Education legislation usually has no sanctions for 
non^cbservance . The CAP, as is the case withyfiost education 

^' * legislation, seldom has penalties for non-compliance* (The 
It,, . penalties, if they exiat, a^e seldom "invoiced. ) The program's 

- success relies upon the acceptance and cooperation of LEAs. If a 
LEA tests, say, only 75 percent of its students, the state h^s no 
viable Remedy or method of encouragement or 'coercion to force 
\ them to test something closer to 100 percent. 
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